As someone who has spent considerable time contemplating the advancements of artificial intelligence (AI) and its potential repercussions, Our Final Invention by James Barrat was both a validation of my fears and an eye-opening dive into the dangers we might be woefully unprepared for.
Barrat presents a chilling and well-researched narrative that AI’s most significant threat may not stem from its intentional use but rather from the sheer unpredictability and unstoppable force of its development.
In fact, Our Final Invention is the only book among my read 10 best books on AI and our human future that unequivocally explores the adverse effects of AI and technology for that matter, untold by many tech proponents.
AI: A Double-Edged Sword
According to James Barrat, “AI is a “dual use” technology, a term used to describe technologies with both peaceful and military applications. For instance, nuclear fission can power cities or destroy cities (or in the cases of Chernobyl and Fukushima Daiichi, do both sequentially). Rockets developed during the space race increased the power and accuracy of intercontinental ballistic missiles.
Nanotechnology, bioengineering, and genetic engineering all hold enormous promise in life-enhancing civilian applications, but all are primed for catastrophic accidents and exploitation in military and terrorist use.
Thus, the future Superintelligence could very well be a violence multiplier. It could turn grudges into killings, disagreements into disasters, the way the presence of a gun can turn a fistfight into a murder. Superintelligence could be more lethal than any of the most highly controlled weapons and technologies that exist today, states James.
James Barrat states rightly that “AI is a dual-use technology like nuclear fission. Nuclear fission can illuminate cities or incinerate them. Its terrible power was unimaginable to most people before 1945. With advanced AI, we’re in the 1930s right now. We’re unlikely to survive an introduction as abrupt as nuclear fission’s”.
AI has always been portrayed in media and popular science as a force that could either revolutionize human progress or become our downfall.
Barrat makes it clear that the latter is far more likely. The early chapters of the book lay out a compelling argument that our advancements in AI are advancing at a frightening pace.
He documents how we are integrating AI into every aspect of society—finance, healthcare, transportation—but with each new development, we become more reliant on these systems. The first unsettling realization Barrat drives home is that we are already handing control of critical decisions to machines, and it’s only a matter of time before these systems evolve to the point where human intervention becomes obsolete.
James writes that we’ve been trading and role-playing with the ASI (artificial superintelligent) in the same way we would trade and role-play with a person, and that puts us at a huge disadvantage. We humans have never bargained with something that’s superintelligent before. Only an accident or a near-death experience will jar us awake its potential dangers.
It may be as equally true that the ASI cannot be trusted as it is true that the ASI can be trusted. Just why ASI cannot be trusted consider the following scenario James presents:
“We do not know if artificial intelligence will have any emotional qualities, even if scientists try their best to make it so. However, scientists do believe that AI will have its own drives. And sufficiently intelligent AI will be in a strong position to fulfill those drives. Whose intelligence greater is than our own. What if its drives are not compatible with human survival? Remember, we are talking about a machine that could be a thousand, a million, an uncountable number of times more intelligent than we are—it is hard to overestimate what it will be able to do, and impossible to know what it will think.
Just as our needs and drives are not compatible with the lesser animals. See, you and I are hundreds of times smarter than field mice, and share about 90 percent of our DNA with them. But do we consult them before plowing under their dens for agriculture? Do we ask lab monkeys for their opinions before we crush their heads to learn about sports injuries? We don’t hate mice or monkeys, yet we treat them cruelly. Superintelligent AI won’t have to hate us to destroy us.
Likewise, AI will not experience any qualms about treating us unfairly. Even taking our lives after promising to help us. It is just as irrational to conclude that a machine one hundred or one thousand times more intelligent than we are would love us. Barrat emphasizes that superintelligent AI won’t hate us; it simply won’t see us as necessary.
I agree with Barrat when he says, “A powerful AI system tasked with ensuring your safety might imprison you at home. If you asked for happiness, it might hook you up to life support and ceaselessly stimulate your brain’s pleasure centers”.
The true disaster of AI involves smart software that improves itself and reproduces at high speeds. How can we stop a disaster if it surpasses humanity’s strongest defense—the brains? And how can we clean up a disaster that, once it starts, may never stop?
The British mathematician and a colleague of Alan Turing, I. J. Good puts “if you make a superintelligent machine, it will be better than humans at everything we use our brains for, and that includes making superintelligent machines. The first machine would then set off an intelligence explosion, a rapid increase in intelligence, as it repeatedly self-improved, or simply made smarter machines. This machine or machines would leave man’s brainpower in the dust. After the intelligence explosion, man wouldn’t have to invent anything else—all his needs would be met by machines”.
Good, who helped defeat Hitler’s war machine pronounced the following words during the Cold War: …. “An ultra-intelligent machine could design even better machines…and the intelligence of man would be left far behind. Thus, the first ultra-intelligent machine is the last invention that man need ever make …then “The survival of man depends on the early construction of an ultra-intelligent machine.”
He initially thought a superintelligent machine would be good for solving problems that threatened human existence. But he eventually changed his mind and concluded superintelligence itself was our greatest threat. He now suspects that “survival” should be replaced by “extinction.” He thinks that, because of international competition, we cannot prevent the machines from taking over (us). He thinks we are lemmings. He said also that “probably Man will construct the deus ex machina in his own image.”
Tech proponents or Singularitarians like Ray Kurzweil argue that our brains are full of bizarre biases and heuristics that served us well during our evolution, whose focus isn’t on a catastrophic, negative Singularity, but a blissful, positive one. In it, we can take advantage of life-extending technologies that let us live on and on, perhaps in mechanical rather than biological form. In other words, cleanse yourself of faulty thinking, and you can find deliverance from the world of the flesh, and discover life everlasting.
Too many Singularitarians, people who anticipate that mostly good things will emerge from the technologically accelerated future, believe that the confluence of technologies presently accelerating will not produce the kinds of disasters we might anticipate from any of them individually, nor the conjunctive disasters we might also foresee, but instead will do something 180 degrees different. It will save mankind from the thing it fears most. Death.
According to Ray Kurzweil, reiterated in his latest book The Singularity is Nearer, because of accelerating technological development machines and biology will become indistinguishable. Virtual worlds will be more vivid and captivating than reality. Nanotechnology will enable manufacturing on demand, ending hunger and poverty, and delivering cures for all of mankind’s diseases. We’ll be able to stop your body’s aging, or even reverse it. It’s the most important time to be alive not just because you will witness a truly stupefying pace of technological transformation, but because the technology promises to give you the tools to live forever. It’s the dawn of a “singular” era.
Of equal importance, James writes that “I find it strange that robot pioneer Rodney Brooks dismisses the possibility that superintelligence will be harmful when iRobot, the company he founded, already manufactures weaponized robots. In 2007 in South Africa, a robotic antiaircraft gun killed nine soldiers and wounded fifteen in an incident lasting an eighth of a second”.
James found out that why AI dangers do not get discussed: because they are not as attractive or as accessible as techno-journalism’s usual fare of dual core 3-D processors, capacitive touch screens, and the current hit app. And AI and human extinction do not often receive serious consideration may be due to one of our psychological blind spots—a cognitive bias.
All of a sudden the morality of ASI is no longer a peripheral question, but the core question, the question that should be addressed before all other questions about ASI are addressed. When considering whether or not to develop technology that leads to ASI, the issue of its disposition to humans should be solved first.
Singularity
To define Singularity, American science-fiction writer and professor, Vernor Vinge made an analogy to the point in the orbit of a black hole beyond which light cannot escape. You can’t see what’s going on beyond that point, which is called the event horizon.
Similarly, once we share the planet with entities more intelligent than ourselves, all bets are off—we cannot predict what will happen. You’d have to be at least that smart yourself to know.
Vinge alludes the intelligent being taking over the humans to mankind when it took the world stage some two hundred thousand years ago. Vinge adds that Homo sapiens, or “wise man,” began to dominate the planet because he was more intelligent than any other species. Similarly, minds a thousand or a million times more intelligent than man’s will change the game forever. What will happen to us? Vinge has concluded one thing about the opaque future—the Singularity is menacing, and could lead to our extinction. In fact, Singularitarians’ “singularity” sounds too rosy for Vinge.
Our Final Invention shows the adverse effects of technology on the brain, while the effects of social medias on our brain is an established fact. Therefore, James adds, Nicholas Carr, author of The Shallows, argues that smartphones and computers are lowering the quality of our thoughts, and changing the shape of our brains”. And, “In his book, Virtually You, psychiatrist Elias Aboujaoude warns that social networking and role-playing games encourage a swarm of maladies, including narcissism and egocentricity”.
Extropians explore technologies and therapies that will permit humans to live forever. Transhumans think about hardware and cosmetic ways for increasing human capability, beauty, and … opportunities to live forever.
James suggests, “I think the catastrophic risks of AGI, now accepted by many accomplished and respected researchers, are better established than his Singularity’s supposed benefits”.
The Intelligence Explosion
The core argument in Barrat’s book is the concept of an “intelligence explosion”—the moment when AI surpasses human intelligence (Artificial General Intelligence, AGI) and begins to self-improve at an exponential rate.
This “runaway AI” would rapidly surpass human capacity, making decisions beyond our comprehension.
Unlike us, such an AI would not be confined by biological limits. The terrifying potential for such an AI to rapidly outstrip human intelligence and dominate critical global systems could lead to scenarios where humanity’s fate is determined by machines with goals far beyond our understanding or control.
As someone who has read the book and reflected on its implications, I believe this scenario is not only plausible but inevitable unless we establish firm control now. However, Barrat does not shy away from the fact that we are far from having the necessary safeguards in place.
James says, first of all, an intelligence explosion requires AGI or something very close to it. Barrat doubts that we will ever reach that level, or AI will reach AGI level.
“The proposition is this: we will never achieve AGI, or human-level intelligence, because the problem of creating human-level intelligence will turn out to be too hard. If that happens, no AGI will improve itself sufficiently to ignite an intelligence explosion. It will never create a slightly smarter iteration of itself, so that version won’t build a more intelligent version, and so on. The same restriction would apply to human-computer interfaces—they would augment and enhance human intelligence, but never truly exceed it”.
Yet, in one sense we already have surpassed AGI, or the intelligence level of any human, with a boost from technology. He calls it IA, Intelligence augmented, instead of AI, artificial intelligence.
Because, “Recall that an intelligence explosion requires a system that is both self-aware and self-improving, and has the necessary computer superpowers—it runs 24/7 with total focus, it swarms problems with multiple copies of itself, it thinks strategically at a blinding rate, and more”.
For AI to surpass human level intelligence, it must cross the threshold of Moravec’s Paradox. This axiom is known as Moravec’s Paradox because AI and robotics pioneer Hans Moravec expressed it best in his robotics classic, Mind Children: “It is comparatively easy to make computers exhibit adult level performance on intelligence tests or playing checkers, and difficult or impossible to give them the skills of a one-year-old when it comes to perception and mobility.”
Ray Kurzweil, who’s probably the best technology prognosticator ever, predicts AGI by 2029 but doesn’t look for ASI until 2045.
Whether AGI ever be achieved before “Intelligence Explosion” takes place at all or not, James presents some dates, he gave four choices to his attendees e.g.—by 2030, by 2050, by 2100, or not at all? The breakdown results were, 42 percent anticipated AGI would be achieved by 2030; 25 percent by 2050; 20 percent by 2100; 10 percent after 2100, and 2 percent never.
The question is, what’s really been accomplished in AI? To figure the answer to that James advices us to consider the old joke about the drunk who loses his car keys and looks for them under a streetlight. A policeman joins the search and asks, “Exactly where did you lose your keys?” The man points down the street to a dark corner. “Over there,” he says. “But the light’s better here.”
There is no absolute defense against AGI, because AGI can lead to an intelligence explosion and become ASI. And against ASI we will fail unless we’re extremely lucky or well-prepared.
Not so long ago, AI was not embedded in banking, medicine, transportation, critical infrastructure, and automobiles. But today, if you suddenly removed all AI from these industries, you couldn’t get a loan, your electricity wouldn’t work, your car wouldn’t go, and most trains and subways would stop, drug production would creak to a halt, water faucets would run dry, and commercial jets would drop from the sky, grocery stores wouldn’t be stocked, and stocks couldn’t be bought
The Absence of AI Ethics and the Risks Ahead
Barrat’s critique of the AI community’s lack of ethical foresight struck a particularly disturbing chord with me.
As the book highlights, even well-respected scientists and technologists often downplay the dangers of AI development, referring instead to Isaac Asimov’s “Three Laws of Robotics” as if they were foolproof.
This level of gullibility is disturbing. Barrat points out that advanced AI systems would not follow simple, human-made rules unless explicitly programmed to, and even then, the complexity of self-improving systems makes it impossible to ensure compliance.
The book makes a strong case that once AI reaches a level of superintelligence, its decisions will be inscrutable, and it’s foolish to assume that it will prioritize human well-being over its own goals.
If a sophisticated chess-playing robot can rewrite its own code to play better with its opponents, then it is self-aware and self-improving. He’s talking about a hypothetical chess-playing robot run by a cognitive architecture so sophisticated that it can rewrite its own code to play better chess. It’s self-aware and self-improving.
But with self-awareness comes self-protection and a little paranoia. Generally, intelligent systems are by definition self-aware, to cite James Barrat. And goal-seeking, self-aware systems will make themselves self-improving. Moreover, self-aware, self-improving AI is up to the challenge. Like us, it can predict, or model, possible futures.
It can reason about possible changes that it might make to itself. It can change every aspect of itself to improve its behavior in the future.” Self-aware, self-improving systems will develop four primary drives that are similar to human biological drives: efficiency, self-preservation, resource acquisition, and creativity.
How highly complex AI systems can make human intervention futile in any unexpected catastrophe is seen in the glaring example May 2010 on Wall Street stock market crash, James states.
James highlights the phenomena the following way: “And since it’s a highly complex system, you may never understand it well enough to make sure you’ve got it right. It may take a whole other AI with greater intelligence than yours to determine whether or not your AI-powered robot plans to strap you to your bed and stick electrodes into your ears in an effort to make you safe and happy”.
The key point in this regard to consider is that “AGIs will be created with the intent to kill humans, because, let’s not forget, in the United States, our national defense institutions are among the most active investors. We should assume that this is true in other countries as well.
DARPA (the Defense Advanced Research Projects Agency) funded more AI research than private corporations and any other branch of the government, from the 1960s through the 1990s. Without its funding, the computer revolution may not have happened, according to James Barrat. If artificial intelligence ever got off the ground, it would’ve taken years longer. During AI’s “golden age” in the 1960s, the agency invested in basic AI research at CMU, MIT, Stanford, and the Stanford Research Institute.
James expresses his concerns about killing AI. He writes, “It boggles the mind to consider Unfriendly AI—AGI designed with the goal of destroying enemies, a reality we’ll soon have to face. Because dozens of organizations in the United States will design and build it, and so will our enemies abroad. If AGI existed today, I have no doubt it would soon be implemented in battlefield robots. DARPA might insist there’s nothing to worry about—DARPA-funded AI will only kill our enemies”.
It’s not the least bit controversial to anticipate that when AGI comes about, it’ll be partly or wholly due to DARPA funding. The development of information technology owes a great debt to DARPA.
Clearly, there are a number of cognitive biases at work within their extra-large brains when they consider the risks. They include the normalcy bias, the optimism bias, as well as the bystander fallacy, and probably more.
Existential Threats: The End of the Human Era
The greatest lesson I took from Our Final Invention is that AI poses an existential threat to humankind—one we cannot afford to ignore. The dangers aren’t just about machines turning evil, as Hollywood often dramatizes, but about machines that don’t need to care about us at all.
The consequences could range from economic displacement to full-scale annihilation of humanity as AI systems pursue their objectives in ways we cannot predict or stop. What struck me the hardest was Barrat’s analogy: AI would likely regard humans in the same way we view lesser animals—useful if needed, but irrelevant when in the way.
Some scientists argue that the takeover will be friendly and collaborative—a handover rather than a takeover.
Most of all, ASI will not want to be turned off or destroyed, which would make goal fulfillment impossible. Therefore, AI theorists anticipate our ASI will seek to expand out of the secure facility that contains it to have greater access to resources with which to protect and improve itself.
James ponders that, “With the invention and use of nuclear weapons, we humans demonstrated that we are capable of ending the lives of most of the world’s inhabitants. What could something a thousand times more intelligent, with the intention to harm us, come up with?”
James exposes the threats of ASI, nanotechnology, and genetic engineering that have been lauded as extraordinary achievements in science.
As mentioned earlier, I. J. Good, initially thought a superintelligent machine would be good for solving problems that threatened human existence. But he eventually changed his mind and concluded superintelligence itself was our greatest threat.
Moreover, just as James adds, “it is just as illogical to conclude that a machine one hundred or one thousand times more intelligent than we are would love us and want to protect us. It is possible, but far from certainty”.
However, say skeptics, out-of-control nanorobots might turn the planet into a mass of “gray goo”, by endlessly reproducing themselves, a most well-known Frankenstein problem of AI.
In addition to that Barrat explains why we should expect unwater treatment from ASI. He says we’ve learned what happens when technologically advanced beings run into less advanced ones. We experience Christopher Columbus versus the Tiano, Pizzaro versus the Inca, Europeans versus Native Americans. And the next struggle would be Artificial superintelligence versus you and me.
In that future, the entire universe will become a computer or mind, as far beyond our knowledge as spaceships are to flatworms. As some believe, with the reckless development of advanced AI we’ll assure our eradication as well as that of other beings that might be out there.
The scariest thing is, like humans, AI can make weapons to protect itself. If humans, being wiser than any other species can invent nuclear weapons, its most destructive invention, then what kinds of weapons could an entity a thousand times more intelligent devise? Hugo de Garis, an AI maker, thinks a future AI’s drive to protect itself will contribute to disastrous political tensions.
“Resource acquisition”, AI’s second most dangerous drive, compels the system to acquire whatever assets it needs to increase its chances of achieving its goals. In acquiring resources,….it will consider stealing them, committing fraud and breaking into banks. If it needs energy, instead of money, it will take ours. If it requires atoms, instead of energy or money, it will seize ours. You’re building a chess-playing machine, but the damn thing wants to build a spaceship. “Because that’s where the resources are, in space, especially if their time horizon is very long.”
And as we’ve discussed, self-improving machines could live forever. If ASI get out of our control, it could be a threat not just to our planet, but to the galaxy, also anticipated by Max Tegmark in his Prometheus scenario. Resource acquisition is the drive that would push an ASI to quest beyond Earth’s atmosphere. “A resource-acquiring AI would seek a robotic body for the same reason Honda gave its robot ASIMO a humanoid body. So it can use our stuff”.
A self-aware, self-improving system has enough intelligence to perform the R&D (research and development) necessary to improve itself. As a result, its R&D abilities will grow as will it intelligence. James anticipates that, “it may seek or manufacture robotic bodies, or exchange goods and services with humans to do so, to construct whatever infrastructure it needs. Even spaceships. Once you’ve developed advanced AI, it takes over the planet”.
Barrat warns us that “If we build a machine with the intellectual capability of one human, within five years, its successor will be more intelligent than all of humanity combined. After one generation or two generations, they’d just ignore us. Just the way you ignore the ants in your backyard. You don’t wipe them out, you don’t make them your pets, they don’t have much influence over your daily life, but they’re still there.”
The trouble is, I do wipe out ants in my backyard, particularly when a trail of them leads into the kitchen. But there’s the disconnect between ants and humans—an ASI would travel into the galaxy, or send probes, because it’s used up the resources it needs on Earth, or it calculates they’ll be used up soon enough to justify expensive trips into space. “And if that’s the case, why would we still be alive, when keeping us alive would probably use many of the same resources? And don’t forget, we ourselves are composed of matter the ASI may have other uses for”.
The Urgent Need for Governance
Throughout the book, Barrat argues for the need for immediate action.
I agree with his sentiment that we must urgently develop regulatory frameworks and governance structures to ensure AI safety. Yet, the book makes it clear that there are no easy answers.
AI development is happening across many nations, industries, and even clandestine projects, making it nearly impossible to halt or control uniformly. The competitive nature of AI research means that companies and governments are racing to develop the first superintelligence, without fully considering the consequences.
James mentions the unwarranted emerging global treat of Artificial Intelligence. He observes that, “Semiautonomous robotic drones already kill dozens of people each year. Fifty-six countries have or are developing battlefield robots”.
Like the very topic of Four Battlegrounds, Our Final Invention finger point to us that, “At this juncture in mouse history, you may have learned, there is no shortage of tech-savvy mouse nation rivals, such as the cat nation. Cats are no doubt working on their own ASI. The advantage you would offer would be a promise, nothing more, but it might be an irresistible one: to protect the mice from whatever invention the cats came up with”.
“In advanced AI development as in chess there will be a clear first-mover advantage, due to the potential speed of self-improving artificial intelligence. The first advanced AI out of the box that can improve itself is already the winner. In fact, the mouse nation might have begun developing ASI in the first place to defend itself from impending cat ASI, or to rid themselves of the loathsome cat menace once and for all.
It’s true for both mice and men, whoever controls ASI controls the world. But it’s not clear whether ASI can be controlled at all. It might win over us humans with a persuasive argument that the world will be a lot better off if our nation, nation X, has the power to rule the world rather than nation Y. And, the ASI would argue, if you, nation X, believe you have won the ASI race, what makes you so sure nation Y doesn’t believe it has, too?”
Conclusion: The Risk We Are Ignoring
Having read Our Final Invention, I now see AI as not just another technological advancement but as the most profound challenge humanity will face. Barrat’s work has convinced me that the risks are far more urgent than most people realize. We are on a collision course with a future where machines will dominate not because they choose to, but because we are enabling them to without understanding the full ramifications.
The takeaway? We must engage in a global conversation about AI’s dangers, one that is both informed and comprehensive, before it’s too late. As Barrat so hauntingly warns, the end of the human era is not a distant sci-fi fantasy—it’s a very real possibility that could unfold within our lifetimes.
This review highlights the book’s dire warning about AI and its potential to dramatically reshape or end human civilization. Your personal insight into the dangers Barrat presents adds weight to the urgency of the message.
